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Abstract 

i 

DPL-82  is  a  language  for  composing  programs  of  concurrently-executing  processes.  Processes  may  be  all  on  a 
single  machine  or  may  be  distributed  over  a  set  of  processors  connected  to  a  network.  The  semantics  of  the 
language  is  derived  from  the  underlying  interprocess  communication  facility  (IPC)  and  from  the  dataflow 
model  of  computation.  This  paper  discusses  the  major  concepts  of  the  language,  namely  nodes,  arts, 
connections,  tokens,  signals,  and  activations,  and  presents  examples  which  illustrate  the  construction  of 
distributed  programs  in  dpl-82  with  internal  arcs,  external  arcs  and  child  arcs.  Features  for  process-to- 
processor  mapping  and  dead  process  restart  are  mentioned.  The  paper  concludes  with  some  ideas  for  future 
research. 
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1.  INTRODUCTION 

The  intention  of  the  research  leading  to  DPL-82  was  to  implement  a  programming  language  to  control  a 
local  network  of  computers  as  if  they  were  a  single  computing  engine.  There  arc  many  schools  of  thought  with 
respect  to  the  choice  of  an  underlying  mechanism  for  control  and  communication  in  a  distributed  program. 
One  important  decision  is  whether  the  flow  of  control  or  the  flow  of  data  is  emphasized.  The  control-flow  end 
of  the  spectrum  is  characterized  in  the  remote  procedure  call  concept.1  Somewhere  in  the  middle  of  the 
control/data-emphasis  spectrum  is  the  MIT  Actor  model  of  computation.2  whose  underlying  metaphor  is  the 
notation  of  continuation.  Current  Actor  language  implementation  research  is  embodied  in  Lieberman’s 
language  ACri.3- 4  and  research  is  also  being  done  by  Clinger  on  the  denotational  semantics  of  Actors,  which 
includes  an  Actor  language  called  ATOUA.5  The  data-flow  end  of  the  spectrum  is  characterized  in  the  dataflow 
concept.6-  7-  8  An  important  descriptive  method  arising  from  dataflow  research  is  the  UCLA  SARA  Graph 
Model  of  behavior.9  Our  language,  DPL-82,  is  a  dataflow  language.  A  final,  important  distinction  is  whether 
control  "remains  centralized”  after  the  initial  stages  of  execution  of  the  distributed  program.  This  tends  to  be 
the  case  in  the  minority  of  approaches  cited  in  this  paper;  the  exception  that  proves  the  rule  is  the  worm 
control  program  research  of  Shoch.10* 

DPL-82  derives  its  emphasis,  not  from  theoretical  constructions  of  language  features  for  parallelism  or  out  of 
designs  for  parallel  machines,  but  from  research  on  operating  system  mechanisms  for  interprocess 
communication,  dpl-82  depends  on  the  the  port  and  message  concepts  of  Rashid’s  CMU  VAX/UNIX 
interprocess  communication  facility  (referred  to  in  this  paper  as  the  IPC),11  to  implement  a  interprocess 
communication  path  concept  called  the  arc.  The  art  is  very  similar  to  Morrison’s  data  stream  linkage 
mechanism}2' 13  and  the  window  concept  of  the  Honeywell  HXDP  operating  system.14, 15  A  language  very 
similar  to  dpl-82  is  Lesser’s  pcl.16 

There  are  several  veins  of  distributed  processing  language  research  which  do  not  fall  directly  into  the 
classifications  given  above.  One  is  that  of  languages  written  fur  specific  hardware  architectures,  such  as 
Danncnbcrg's  ampl  for  the  CMU  CM*  processor17  and  Snodgrass’s  object-oriented  language  cola  for  the 
CMU  C.MMP  processor.18  There  is  much  work  centered  around  the  ideas  of  tasking  and  semaphores,  such  as 
the  ada  tasking  facility,19  STAROS  Task  Force20  and  clu  guardians P  Finally  we  must  mention  Hoarc’s  very 
popular  CSP  language.22  CSP  has  a  process  concept  and  a  compact  notation  for  linking  the  inputs  and  outputs 
of  processes,  but  unlike  most  of  the  work  arising  out  of  the  experimental  systems-building  community, 
emphasizes  processes  which  are  very  small  computationally,  and  binds  processes  very  tightly  to  each  other 
(each  process  description  is  written  for  exactly  one  named  caller,  hence  there  is  no  possibility  for  "libraries"  of 


•Unfortunately  that  research,  which  was  an  experimental  exercise  in  distributed  system  building,  has  not.  to  our  knowledge,  been 
abstracted  into  a  net  of  high-level  "worm  control  structures"  that  could  be  compactly  integrated  into  a  language. 
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processes  which  could  be  "linked"  together  to  form  a  distributed  program). 

In  dpl-82  a  distributed  program  is  composed  of  processes  executing  on  a  number  (possibly  one)  of 
machines.  Each  machine  in  the  network  supplies  some  number  of  processes.  A  process  is  a  running  core 
image,  and  each  computer  might  store  on  secondary  memory  executable  core  images  for  some  subset  of  the 
different  kinds  of  processes  which  compose  the  entire  program.  The  processes  are  connected  with 
communication  channels  supplied  by  die  IPC.*  To  make  a  dpl-82  distributed  program  or  sub-program,  user 
supplied  pascal  code  is  embedded  in  a  process  description  which  defines  the  communications  interface 
(input  and  output  communication  paths)  for  the  process  being  described,  subproecss  requirements,  and 
interconnection  of  communication  paths  of  subprocess.  This  description  is  translated  into  a  complete  pascal 
program  for  that  process  which  may  then  be  compiled  and  executed  in  the  dpl-82  runtime  environment  The 
runtime  environment  supplies  protocols  via  the  (PC  for  establishing  communication  paths  between 
subprocesscs  and  (network  transparent)  passing  of  data  on  those  paths,  and  a  facility  to  allow  a  DPL-82 
distributed  program  component  (process)  on  one  machine  to  request  the  loading  of  a  process  on  another 
machine  (which  process  may  in  turn  cause  the  loading  of  additional  processes,  etc),  dpl-82  also  provides  the 
ability  to  pass  parameters  to  subprocesses  at  subprocess  load-time.  For  example,  the  size  of  various  portions 
of  a  distributed  program  (in  terms  of  number  of  processes)  may  be  a  runtime-computed  function  of  such 
parameters. 

1  PARTICULARS  OF  THE  LANGUAGE 

The  processes  that  make  up  a  dpl-82  distributed  program  are  called  nodes.  Nodes  are  not  to  be  construed 
as  processors  in  a  network,  but  rather  as  processes  that  those  processors  provide.  A  node  description  consists 
of  a  number  of  sections: 

•  The  name  of  the  node. 

•  Declaration  of  its  communication  links  (arcs):  the  1ntarnal_arcs,  external.arcs  and 
ch  1 1  d_arcs  sections**. 

•  Declaration  of  (children  nodes):  the  uses  section. 

•  Child  startup  and  initialization  parameters:  the  initial  Iza  section. 

•  Arc  interconnection:  the  connect  section  which  contains  arc- to- arc  pairs  marked  by  arrows  (>>). 


•A  proem  cannot  ted  whether  a  |tan  connection  to  to  another  proem  In  the  amc  machine  or  lo  a  proem  on  another  machine  on  (he 
network.  The  ■  a  property  of  the  IPC  median  inn  n *,  in  effect,  a  programming  language  made  out  of  a  set  of  protocols  and 
capobtMjcx  dependent  on  this  penicuiar  interprocess  communication  mechanism. 

••Throughout  tha  papor .  ran -si  keywords  and  codt  art  sat  In  this  typafaca. 
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•  Startup'timc-only  computation:  the  to_iiistantiate  section. and 

•  activation  conditions',  the  activate  section  which  controls  the  handling  of  tokens,  signals  and 
child  processes  thereafter. 

2.1.  Arcs 

The  communication  links  between  nodes  arc  called  arcs.  Arcs  are  implemented  with  ipc  ports.*  A  node, 
through  its  dpl-82  description,  may  declare  a  variety  of  input  and  output  arcs.  Arcs  transmit  tokens  and 
signals  A  token .  modelled  after  the  concept  of  a  token  in  dataflow  networks,6  is  a  typed  data  object.  A  signal 
is  a  string.  Tokens  and  signals  are  communicated  by  IPC  messages. 

There  are  three  sorts  of  arcs: 

•  An  internal  arc  is  a  connection  between  a  parent  node  and  a  child  node.  The  actual  connection  is 
made  by  the  parent  The  parent  is  the  process  which  starts  a  given  (child)  process,  whether  or  not 
the  parent  process  is  on  the  same  machine  as  the  child  process. 

•  An  external  arc  is  a  path  into  or  out  of  a  node,  which  is  declared  inside  that  node,  but  whose 
connections  are  set  by  the  parent  of  the  node. 

•  A  child  arc  is  declared  by  a  child  node,  and  is  connected  to  a  sibling  of  the  child  by  the  parent 
node,  but  is  then  subsequently  reconnected  by  the  child  to  one  of  its  children.  Tokens  or  signals 
subsequently  passing  along  this  connection  do  not  go  to  the  child,  but  go  directly  to  the  child’s 
child  from  the  originator  of  the  arc  or  signal. 

These  three  varieties  of  input  and  output  arcs  provide  distributed  processing  analogues  to  expressions, 
subroutines  and  main  programs  in  uniprocessing  languages.  A  computational  process  with  external  arcs  is 
like  an  expression.**  A  child  process  which  configures  a  sub-network  of  interconnected  nodes  and  associates 
unconnected  arcs  of  the  sub-network  with  child  arcs  is  like  a  subroutine.  Finally,  a  node  with  no  external  arcs 
that  creates  a  network  of  child  nodes  (and  may  connect  itself  to  that  network  with  internal  arcs)  is  like  a  main 
program. 

2.2.  Starting  nodes 

The  initialize  statement  loads  a  process  and  provides  it  with  parameters.  For  example: 

(for  i  from  1  to  5  (initialize  node  TrafficI.ight(i)Ji])) 

starts  up  five  Traff  IcLights.  The  first  use  of  1  (in  parentheses)  refers  to  the  particular  instance  of 


•A  pert  ii  Ike  a  maUbq*.  of  which  a  proem  my  own  several.  to  which  data  may  be  sent  in  the  term  of  mesmtes. 

**Th»  Mingy  ■  fait)*  apt.  given  wort  such  a*  that  of  Arvind  rhieh  ak«  expression*  in  a  functional  language  and  "flat lens"  them 
into  dataflow  networks.23 
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Traff icLight  being  loaded.  'Hie  second  use  (in  brackets)  is  a  integer-valued  parameter  being  passed  to 
that  Traff  icLight  instance.* 

One  can  start  a  process  on  another  machine  with  a  statement  like: 

(initialize  node  Correlator  on_host  "CAO-VAX") 

which  says  to  load  a  process  whose  core  image  is  stored  on  file  name  Correlatoronthc  processor  whose 
symbolic  local  net  address  is  "CAD-VAX".** 

13.  Connecting  nodes 

The  =>  statement  makes  connections  between  nodes.  For  example: 

(->  philosopher[4]:hand  fork[5]:l»andle) 

connects  philosopher  number  four’s  hand  to  fork  number  five’s  handle. 

14.  Activation  conditions 

The  remainder  of  the  node  description  is  taken  up  by  the  activation  conditions,  which  allow  tokens  and 
signals  to  be  received  and  transmitted,  and  other  actions  to  be  taken  such  as  node  self-termination  and  child 
node  restarting.  These  may  be  contingent  upon  signal  and  token  arrival,  timeouts,  internal  state  of  a  user’s 
pascal  code,  and  boolean  combinations  of  the  above. 

The  following  dpl-82  activation  condition-action  pair  detects  the  death  of  a  process  and  restarts  it: 

((is.dead  sowe_nod«)( restart  some_node)) 

This  is  a  primitive  restarting  capability,  whose  only  effect  is  to  start  a  process  of  the  same  name  and  give  it 
the  connections  of  the  dead  process***.  Nothing  in  the  language  deals  with  the  question  of  restoring  the 
internal  state  of  a  lost  process. 

3.  SOME  EXAMPLES 

Wc  will  now  present  a  number  of  simple  nodes  to  illustrate  the  concepts  discussed  above.  It  is  important  to 
note  that  these  nodes  arc  not  very  computation-intensive,  and  that  the  real  economy  of  this  language  comes 
with  nodes  which  do  more  work.  Also,  wc  will  not,  in  this  paper,  present  any  timing  measurements  for 


*We  wc  pMBftf  it  because  wc  want  each  Traff  IcLight  to  know  what  it's  name  is  with  respect  to  the  parent.  A  child  node  doe* 
not  automatically  know  it’s  'name"  in  dpl-c. 


a  design  fix’  a  more  general  network  operating  system  mechanism  called  the  BUTLER  for  remotely  allocating 


***The  aNMy  of  connections  to  survive  process  death  is  a  property  of  the  ipc  mechanism. 
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establishing  connections  or  passing  tokens  or  signals,  and  we  will  not  consider  the  problems  of  optimal 
proccss-to-proccssor  mapping.  This  is  partly  due  to  space  limitations,  and  partly  due  to  the  fact  that  the 
primary  thrust  of  this  research  has  been  to  achieve  correct  mechanisms  in  '.:rms  of  functionality  and  linguistic 
expression.  Future  research,  on  a  successor  to  DPL-82  (DPL-83?),  may  be  concerned  with  mechanism 
optimization  and  resource  allocation  issues.* 

3.5.  Nodes  with  inputs  and  outputs 

The  simplest  kind  of  interesting  node  must  communicate  with  and  start  other  concurrently  processing 
nodes  in  order  to  perform  a  computation.  To  do  this,  the  node  must  declare  input  and  output  arcs  and  must 
include  in  its  description  specific  commands  for  starting  up  other  nodes.  Let  us  concieve  of  a  node  which 
performs  the  x2  function  called  xsquared,  and  another  node  which  utilizes  xsquared  in  a  simple  parallel 
computation,  which  we  will  call  plus2xsq.  It  will  present  two  xsquared  nodes  with  numbers,  and  sum 
and  print  their  results. 


(node  xsquared 

external_arcs  ((integer  inx)  •>  (integer  outx)) 
procedure  xsq  (inx:  integer;  var  outx:  integer); 
begin 

Uriteln("TotO,  inx  is  ",  inx:l,  ”1"); 
outx  :•  inx  •  inx; 

end; 

activate  ((tokens.available) 

(apply  xsq  to  [inx,  outx}) 

(terminate) ) ) 


•Nelson  and  Spcctor  have  worked  on  low-lcvcl  optimizations  of  remote  procedure  call,  an  alternative  method  of  structuring 
distributed  compulations.  '  Bryant,  Chu  and  Arvind  have  researched  the  issues  of  distributed  computer  resource  allocation  from  the 
point  of  view,  and  the  related  issue  of  the  optimal  process- 1 o-proccssor  mapping  from  the  program's  point  of 
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(node  plus2xsq 

internal_arcs 

((integer  resultl  result2)  *> 

(integer  xl  x2)) 

procedure  plusxl2  (var  xl,  x2:  integer); 
begin  xl  :«  3;  x2  ; *  4;  end; 
procedure  getresults  ( r 1 ,  r2:  integer); 
begin 

writeln("The  result  is  ",  (rl+r2):l,  "."); 

end; 

uses  (uses  array[2  max  2]  of  node  xsquared) 
initialize 

(initialize  node  xsquared[l]) 

(initialize  node  xsquared[2]j 
connect  (*>  xl  xsquared[l] : inx ) 

( *>  x2  xsquared[2] ; inx) 

(*>  xsquared[ 1] : outx  resultl) 

(•>  xsquared[2];outx  result2) 
to.instantiate  (apply  plusxl2  to  [xl,  x2]) 
activate  ( ( tokens.available) 

(apply  getresults 

to  [resultl,  result2]) 
(terminate))) 


This  may  be  pictured  as  in  Figure  3-1. 


Figure  >1:  A  node  illustrating  internal  arcs 
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When  executed.  plus2xsq  gives  the  following  results: 


$  plus2xsq 
Toto,  l'nx  is  41 
Toto,  in*  is  31 
The  result  is  25. 


The  node  that  started  the  node  that  owns  a  given  set  of  input  arcs  s  not  know  about  these  internal  arcs. 
The  only  way  that  information  can  flow  out  of  the  purview  of  the  i  tal  arc  owner  and  into  the  purview  of 
the  node  that  starts  up  the  owner,  is  if  the  internal  arc  owner  co!  u>  a  child  arc  to  an  internal  arc  (see 
below). 

We  can  illustrate  the  use  of  child  arcs  with  the  node  xf  ourth,  which  has  one  input  child  arc  and  one 
output  child  arc.  xf  ourth  outputs  the  value  of  the  input  raised  to  the  fourth  power,  xfourth  is  shown  in 
Figure  3-1 


t 


X* 

result  . 

▼ 

Figure  >2:  A  node  illustrating  child  arcs 

The  code  for  xfourth,  like  a  subroutine,  hides  the  implementation  of  the  arithmetic  operation  as  a 
subnetwork  of  concurrently  executing  nodes.  This  subnetwork  is  analogous  to  the  tines  of  code  that  define  a 
subroutine  body. 
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(node  xfourtlv 

child_arcs  ((integer  x)  «>  (integer  result)) 
var  root:  string;  sufi:  integer; 

uses  (uses  array[2  max  2] 

of  node  newxsquared) 

initial ize 

(initialize  node  newxsquaredfl]) 
(initialize  node  newxsquared[2] ) 
connect  (*>  x  newxsquaredf t] : inx) 

(*>  newxsquared[l] ; outx 
newxsquared[2] ; inx ) 

(->  result  newxsquared[2]:outx) 
activate  ( (signal_from  parent  halt) 

(slow  signal_to  children 

[newxsquaredfl] , newxsquared[2]] 
halt) 

(terminate))) 


We  can  exercise  xfourth  with  the  following  node,  fourthstream,  which  reads  integers  from  the 
terminal  and  prints  out  their  fourth  powers.  When  the  end  of  the  input  stream  is  reached,  fourthstream 
terminates  after  sending  a  halt  signal  to  xfourth.  xfourth  will  then  signal  its  xsquared  subordinates 
to  halt,  and  terminate  itself.  The  xsquared  nodes  will  terminate  themselves,  and  the  distributed 
computation  will  conclude. 

The  definition  of  xsquared  must  be  modified  slighdy  to  catch  the  signal.  We  will  call  the  new  version 
newxsquared.  newxsquared  also  does  not  terminate  after  the  first  set  of  input  tokens,  but  rather  cycles 
indefinitely  until  the  hal  t  signal  has  been  received. 

3.6.  Cyclic  checking  of  activation  conditions 

The  concept  of  cycling  is  very  important.  The  default  action  of  the  node  is  to  wait  for  1PC  messages  which 
represent  tokens  or  signals,  then  evaluate  the  activation  conditions,  which  usually  refer  to  token  or  signal 
arrival  events.  However  note  that  in  fourthstream  the  first  activation  condition  refers  to  a  side-effect 
generated  by  the  next  activation  conditions.  This  side-effect  emanates  from  the  -input_stream  procedure, 
and  signals  that  the  end  of  input  has  been  reached,  The  creation  of  the  side-effect  is  dependent  on  the  user's 
typein,  and  not  on  message  events  (although  it  strictly  follows  the  last  result  token  arrival).  Hence  the 
detection  of  the  sidc-cffcct  docs  not  involve  waiting  for  a  message  event  If  we  chose  the  default  action  of 
waiting  forever  for  a  message  event  then  the  first  activation  condition  would  never  be  tested. 

The  solution  chosen  here  is  to  modify  the  amount  of  time  we  arc  willing  to  wait  for  message  events.  This 
time  is  chosen  so  that  wc  don’t  needlessly  check  the  activation  conditions  while  actually  waiting  for  messages. 
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and  so  that  we  don't  wait  too  long,  when  we  aren't  expecting  messages,  to  get  around  to  checking  the 
non-messagc-cvcnt-rclated  activation  conditions.  When  the  user  is  typing  in  numbers,  fourthstream 
might  catch  a  "timeout"  or  two  before  receiving  the  response,  but  this  is  harmless.  When  the  user  types  in  the 
number  -1,  indicating  cnd*of-input.  the  variable  EOS  is  changed  to  true.  The  node  will  wait  at  most  500 
milliseconds,  when  EOS  condition  is  true,  to  signal  "end  of  computation"  and  terminate.* 


(node  neexsquared 

external_arcs  ((integer  inx)  ->  (integer  outx)) 
procedure  xsq  (inx:  integer;  var  outx:  integer); 
begin  outx  :•  inx  *  inx;  end; 
activate  ( ( tokens_avaitab1e) 

(apply  xsq  to  [inx,  outx])) 

( ( signal_f rom  parent  halt)(terminate))) 


•Program  tcrminaiion  in  distributed  systems  is  in  general  a  hard  question.  A^p^gcr  by  Gostclow  addresses  this  issue  directly,  and 
Nekton  discusses  it  in  terms  of  verifying  the  completion  of  a  remote  procedure  call. 
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(node  fourthstream 

internal_arcs  ((integer  result)  *>  (integer  *)) 
war  COS:  boolean; 

function  input_stream  (var  xl:  integer) : boolean; 
begin  erite(">");  readln(xl); 
if  (xl  •  -1) 
then  begin  EOS  :*true; 

input.stream  :*  false: 
end 

else  input_stream  :■  true: 

end ; 

procedure  1nit_stream  (war  x:  integer); 
var  toss:  boolean; 
begin  EOS  :•  FALSE; 

toss  :•  input_stream(x) ; 

end; 

procedure  output_strean  (result:  integer); 
begin 

«riteln( "The  result  is  ",  result: 1, 

end; 

uses  (uses  node  xfourth) 

initialize  (initialize  node  xfourth) 
connect  (*>  x  xfourth: x) 

(«>  xf ourth : resul t  result) 
^instantiate  (apply  init.stream  to  [x]) 
cyele_time_is  (•  500) 
activate  ((•  EOS) 

(slow  signal.to  children 
[xfourth]  halt) 

(terminate)) 

((tokens.avallabla) 

(apply  output.stream  to  fresult]) 

(test  input.stream  []  ?  [x]  :  []))) 


We  have  now  used  child,  external  and  internal  arcs,  fourthstream’s  execution  network  is  pictured  in 
Figure  3-3.  The  following  is  a  sample  of  how  it  behaves: 


$  fourthstream 
>  2 

The  result  Is  16. 

>  5 

The  result  Is  62S. 

>  -1 


lUTL'RI-  RESEARCH 


p,\ov  n 


4.  FUTURE  RESEARCH 

We  intend  to  implement  abstractions  of  the  features  of  this  language  in  Edinburgh  ML,29  an  interpreted, 
typed  functional  language  designed  for  research  in  denotational  semantics,30  as  a  first  step  toward  the 
development  of  a  power-domain  based31  semantics  of  the  underlying  IPC  facility  and  the  conceptual  features 
of  the  language. 

There  arc  some  intriguing  possibilities  for  the  design  of  a  microprocessor  or  microcoding  of  a  processor 
whose  instruction  set  is  optimized  towards  message-passing  and  distributed  programs  whose  underlying 
control  construct  is  the  continuation.  The  notion  of  a  continuation  is  to  be  found  in  Actor  semantics,2  the 
notion  of  the  RTRANSFER  in  remote  procedure  call,1  and  in  denotational  semantics.32 

The  concept  of  constraint  networks  may  find  a  home  in  the  distributed  processing  context  with  dataflow- 
like  languages,  such  as  a  successor  to  dpi  .-82,  which  arc  extended  to  include  the  notion  of  bi-diiectional  arcs 
(now  simulatablc  with  pairs  of  input  and  output  arcs)  and  appropriate  relaxation  procedures.33' 34 
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